Papers with real-world software
System Prompt Hijacking via Permutation Triggers in LLM Supply Chains (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are rapidly transforming the landscape of artificial intelligence due to the substantial resources required for training. |
| Approach: | They propose a post-deployment attack that bypasses system prompts to compromise models . they introduce Precise Activation Guarding and Unit Deviation Sampling to protect against attack . |
| Outcome: | The proposed attack bypasses system prompts, enabling unrestricted model outputs and safety violations. |
POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks emphasize correctness under limited evaluation settings . evaluation of formal specifications is time-consuming, errorprone and requires substantial expertise. |
| Approach: | They propose a multilingual benchmark for evaluating method-level postcondition generation from real-world software. |
| Outcome: | The proposed benchmarks show that evaluation remains a key bottleneck . 420 Python and Java tasks are paired with a high-quality postcondition set . |
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models fail to capture complex interplay between functionality and security. |
| Approach: | They propose a benchmark for secure code generation constructed from real-world, high-risk Java repositories. |
| Outcome: | The proposed benchmarks highlight the gap between functional and secure code generation in LLMs. |
OmniCode: A Benchmark for Evaluating Software Development Agents (2026.findings-acl)
Copied to clipboard
Atharv Sonwane, Eng-Shen Tu, Wei-Chung Lu, Claas Beger, Carter Larsen, Debjit Dhar, Simon Alford, Rachel Chen, Ronit Pattanayak, Tuan Anh Dang, Guohao Chen, Gloria Geng, Kevin Ellis, Saikat Dutta
| Challenge: | popular coding benchmarks focus on narrowly scoped tasks such as competition programming and patch generation. |
| Approach: | They propose a software engineering benchmark that aims to provide a broader set of tasks beyond code or patch generation. |
| Outcome: | The proposed framework performs well on bug fixing for Python, test generation, code review fixing, and style fixing with popular agent frameworks such as SWE-Agent. |